Two main types of three-dimensional computer data models of the real world are in use today. One deals with the real world by sampling it, while the other approximates the real world by approximating it using mathematical functions. The sampling method leads to volumetric models, and uses voxels as the main representation unit. The mathematical method, an example of which is called CAD (Computer Aided Design), creates models which use polygons as primitives. Polygons are good for rendering the external surfaces of a three-dimensional object, but accurately rendering the internal features of the human body, for example, requires so-called “volume rendering” using “voxels”, which are three-dimensional image components.
Specifically, volume rendering is a technique for visualizing three-dimensional arrays of sampled data. Examples of sampled three-dimensional data are medical data from CAT or MRI scanners, seismic data, or any other volumetric information for which geometric surfaces are difficult to generate or unavailable. Volume rendering takes the original sampled data, interpolates between the available data to fill in the gaps, and displays the resulting three-dimensional image to the user on a screen (using perspective or orthogonal projection).
A major issue when interacting with typical images produced by volume rendering is the sheer size of the data sets. More often than not, the data sets sources like CT and MR) need to be processed at the same time. This factor, together with other factors such as the amount of interpolation which needs to be done and the footprint of the rendered volume, can adversely affect the rendering speed. This problem is compounded when the user tries to view the volume at a high magnification, during which the system can slow to a crawl.
By contrast, mathematical methods such as CAD represents three-dimensional objects as mathematical functions, usually polygons and polylines. However, as for volume rendering, the sheer size of the data makes real-time interaction a problem because of the rendering speed required to produce new images in less than 100 milliseconds (10 per second).
Both volume rendering and CAD result in a rendering in a “frame buffer”
With the increasing power of computers, volume rendering and CAD are capable of generating increasingly complex data models. However, the computers' ability to “render” the models is limited by several factors:                The size of the model (i.e. the number of elements in it, defined as the number of polygons in the case of CAD, or the number of voxels in the case of volume rendering). The bigger the size, the slower it is to process and render on the screen.        The size of the footprint (i.e. the area covered by the projection of the object on the frame buffer of a screen) when rendered. The larger the footprint, the slower the processing.        The degree of simplification when rendering the model (i.e. the level of detail). Objects can be simplified in their final display to speed up rendering.        
One existing technology for displaying three dimensional models (whether created by volume rendering or CAD) is called the Dextroscope, which is used for visualisation by a single individual. A variation of the Dextroscope, for use in presentations to an audience, and even a large audience, is called the DextroBeam. This Dextroscope technology displays a high-resolution stereoscopic virtual image in front of the user.
The software of the Dextroscope uses an algorithm having a main loop in which inputs are read from the user's devices and actions are taken in response. The software creates a “virtual world” which is populated by virtual “objects”. The user controls a set of input devices with his hands, and the Dextroscope operates such that these input devices correspond to virtual “tools”, which can interact with the objects. For example, in the case that one such object is virtual tissue, the tool may correspond to a virtual scalpel which can cut the tissue.
FIG. 1 shows the stages in the operation of the Dextroscope. There are three main stages: (1) Initialization, in which the system is prepared, followed by and endess loop of (2) Update, in which the input from all the input devices are received and the objects are updated, and (3) Display, in which each of the updated objects in the virtual world is displayed in turn.
Within the Update stage, the main tasks are:                reading all the input devices connected to the system.        finding out how the virtual tool relates to the objects in the virtual world        acting on the objects according to the programmed function of the tool        updating all objects        
The tool controlled by the user has four states: “Check”, “StartAction”, “DoAction” and “EndAction”. Callback functions corresponding to the four states are provided for programming the behaviour of the tool.
“Check” is a state in which the tool is passive, and does not act on any object. For a stylus (a three-dimensional-input device with a switch), this corresponds to the “button-not-pressed” state. The tool uses this time to check the position with respect to the objects, for example if is touching an object.
“StartAction” is the transition of the tool from being passive to active, such that it can act on any object. For a stylus, this corresponds to a “button-just-pressed” state. It marks the start of the tool's action, for instance “start drawing”. DoAction is a state in which the tool is kept active. For a stylus, this corresponds to “button-still-pressed” state. It indicates that the tool is still carrying out its action, for instance, “drawing”. EndAction is the transition of the tool from being active to being passive. For a stylus, this corresponds to “button-just-released” state. It marks the end of the tool's action, for instance, “stop drawing”.
A tool is typically modelled such that its tip is located at object co-ordinates (0,0,0), and it is pointing towards the positive z-axis. The size of a tool should be around 10 cm. A tool has a passive shape and an active shape, to provide visual cues as to which states it is in. The passive shape is the shape of the tool when it is passive, and active shape is the shape of the tool when it is active. A tool has default passive and active shape.
A tool acts on objects when it is in their proximity. A tool is said to have picked the objects.
Generally, a tool is said to be “in” an object if its tip is inside a bounding box of the object. Alternatively, the programmers may define an enlarged bounding box which surrounds the object with a selected margin (“allowance”) in each direction, and arrange that the software recognises that a tool is “in” an object if its tip enters the enlarged bounding box. The enlarged bounding box enables easier picking. For example, one can set the allowance to 2 mm (in the world's coordinate system, as opposed to the virtual world), so that the tool will pick an object if it is within 2 mm of the object's proximity. The default allowance is 0.